Image projection apparatus

ABSTRACT

An image projection apparatus includes a detector that detects a first region emitting non-visible light; a projector that projects a visible-light image onto a second region including the first region detected by the detector; and a controller configured to generate the visible-light image. The controller varies colors of pixels of the visible-light image in a stepwise manner depending on emission intensities of the non-visible light at positions corresponding to the pixels within the first region.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of PCT/JP2019/012131 filedMar. 22, 2019, which claims priority to Japanese Patent Application No.2018-072647, filed Apr. 4, 2018, the entire contents of each of whichare incorporated herein by reference.

BACKGROUND 1. Technical Field

The present disclosure relates to an image projection apparatusprojecting a visible-light image onto a region from which non-visiblelight emission has been detected.

2. Related Art

JP 9-24053 A (Document 1) discloses a surgical operation support systemallowing a fluorescent imager to output image data indicative of anaffected area of a living body subjected to a surgical operation andreproducing an image based on the above image data by an imageprojector, for display on the actual affected area. A substance emittingfluorescent light by irradiation of light of a predetermined wavelengthis applied in advance to the living body affected area. That is, thissystem supports confirmation of lesions by displaying a fluorescenceimage fluoresced from the affected area onto the actual affected area.

WO2016/157260 (Document 2) discloses a visible-light projection deviceincluding a detector detecting a first region emitting non-visiblelight, a projector performing projection using visible light, onto asecond region including the first region detected by the detector, and acontroller causing the projector to perform projection, based on colorselected by the operator. If first color is selected as color of visiblelight projected by the projector onto a region of the second regionother than the first region, the controller informs the operator thatcolor of visible light projected by the projector onto the first regioncan be selected from among first options. On the other hand, if secondcolor different from the first color is selected as color of visiblelight projected by the projector onto a region of the second regionother than the first region, the controller informs the operator thatcolor of visible light projected by the projector onto the first regioncan be selected from among second options different from a candidatecombination included in the first options. Document 2 further disclosesthat in the case of allowing the projector to perform projection coloredin multi-gradations onto the first region depending on the intensity ofnon-visible light from portions making up the first region detected bythe detector, the saturation of color of visible light projected by theprojector is set differently depending on the gradations. Such aconfiguration enables a video projection with a high visibility to beimplemented.

SUMMARY

The present disclosure provides an image projection apparatus capable ofimplementing a video projection with a high visibility.

An image projection apparatus according to the present disclosureincludes: a detector that detects a first region emitting non-visiblelight; a projector that projects a visible-light image onto a secondregion including the first region detected by the detector; and acontroller configured to generate the visible-light image. Thecontroller varies colors of pixels of the visible-light image in astepwise manner depending on emission intensities of the non-visiblelight at positions corresponding to the pixels within the first region.

According to the present disclosure, an image projection apparatuscapable of projecting a high-visibility image can be provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view explaining a configuration of a surgery support systemof an embodiment of the present disclosure.

FIG. 2 is a block diagram showing a configuration of a control device ofthe surgery support system.

FIG. 3 is a flowchart showing actions of the surgery support syste.

FIG. 4 is a view explaining the contents of color conversion processingin a display mode and projection modes by the surgery support system.

FIG. 5 is a view explaining an HSV color space.

FIG. 6 is a view explaining images of an image displayed or projected ineach of the display mode and the projection modes.

FIG. 7A is a view explaining an image of an infrared image imaged by aninfrared camera.

FIG. 7B is a view explaining an image of a projection image projected ina third projection mode (multi-colored projection mode), onto theinfrared image shown in FIG. 7A.

FIG. 8 is a view explaining the range of hue of projection image colorsset in the third projection mode.

FIG. 9 is a flowchart showing the color conversion processing in theprojection modes.

FIG. 10 is a view explaining the contents of color conversion processingin a black-and-white inverted version in the display mode and theprojection modes by the surgery support system.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of an image projection apparatus of the present disclosurewill now be described in detail with appropriate reference to thedrawings. Note, however, that unnecessarily detailed explanations may beomitted. For example, detailed explanations of already well-knownmatters or duplicate explanations for substantially the sameconfigurations may be omitted. This is for the purpose of avoidingunnecessarily redundancy in the following description to facilitate theunderstanding of those skilled in the art.

It is to be noted that the applicant provides the accompanying drawingsand the following description in order for those skilled in the art tofully understand the present disclosure but do not intend to therebylimit subject matters described in claims. Hereinafter, an example willbe described where a projection system of the present disclosure isapplied to a surgery support system used in hospitals.

First Embodiment 1. Overview of Surgery Support System

FIG. 1 is a view explaining a configuration of a surgery support systemthat is an embodiment of an image projection system of the presentdisclosure. As shown in FIG. 1, a surgery support system 100 is disposedvertically above a patient 120 lying on a surgical table 110 andincludes an image projection apparatus 200 that projects an image ontoan affected area of the patient.

Before surgery utilizing the surgery support system 100, aphotosensitive substance is administered in blood, etc. of the patient120 undergoing surgery. The photosensitive substance is a substance thatreceives excitation light and emits fluorescent light. In thisembodiment, indocyanine green (hereinafter, referred to as “ICG”) isused as an example of the photosensitive substance. ICG is a medicallyapproved reagent that can be used on the human body. When administeredinto blood, ICG accumulates on the affected area 130 where blood orlymph flow is disrupted. When irradiated with infrared excitation lightaround 800 nm, ICG emits infrared fluorescent light of a wavelengtharound 850 nm of peak wavelength. Accordingly, if a region (hereinafter,referred to as “ICG light-emitting region”) emitting infraredfluorescent light can be detected, it becomes possible to identify aregion of the affected area 130.

The surgery support system 100 detects the ICG light-emitting region toidentify the region of the affected area 130. To allow the identifiedregion of the affected area 130 to be visible to the doctor, the surgerysupport system 100 projects a visible-light image generatedcorrespondingly to the identified region of the affected area 130, as aprojection image, onto a region including the ICG light-emitting region,in a superimposed manner. In particular, the visible-light imageincludes an image having a region corresponding to the ICGlight-emitting region colored with a high visibility color. By thisprojected visible-light image, the surgery support system 100 cansupport doctor's region identification (visual recognition) of theaffected area 130 when performing surgery on the affected area 130.

In particular, the surgery support system 100 of this embodiment canproject with the visible-light image (projected image) region colorsdiffering depending on the emission intensity of the ICG light-emittingregion. That is, ICG emission intensity distribution (ICG concentrationdistribution) can be projected in multiple colors, enabling the doctor,etc. to visually easily recognize the ICG concentration distribution. Byallowing the ICG concentration distribution to be visually recognized inthis manner, it becomes possible to easily confirm the range of theaffected area 130 and the depth of a location where the affected area130 is present in organ,

2. Configuration of Surgery Support System

Configuration details of the surgery support system 100 will bedescribed. The surgery support system 100 is disposed and used in anoperating room of a hospital. As shown in FIG. 1, the surgery supportsystem 100 includes the image projection apparatus 200, a control device300, and an infrared excitation light source 230. The surgery supportsystem 100 includes a display 160 and an operating part 170, foraccepting various setting operations from an operator 140.

Although not shown, the surgery support system 100 includes a mechanism(a driving arm mechanically connected to the image projection apparatus200, casters of a pedestal on which a set of the surgery support system100 is mounted, etc.) for changing a position to dispose the imageprojection apparatus 200.

The image projection apparatus 200 is an apparatus that integrallyencompasses imaging means and irradiation means. The image projectionapparatus 200 includes an infrared camera 210, a dichroic mirror 211,and a projector 220. The image projection apparatus 200 detects infraredfluorescent light 223 issued from the patient 120 lying on the surgicaltable 110. The image projection apparatus 200 then irradiates visibleirradiation light 222 from the projector 220 onto a region including theregion (ICG light-emitting region) of the affected area 130 indicated bythe detected infrared fluorescent light 223. In order to detect theinfrared fluorescent light 223 more properly and to irradiate thevisible irradiation light 222 more properly, it is preferred that theimage projection apparatus 200 be disposed right above the patient 120lying on the surgical table 110.

The control device 300 is a device that overall controls actions ofparts making up the surgery support system 100. The control device 300connects to and controls the infrared camera 210, the projector 220, andthe infrared excitation light source 230.

FIG. 2 is a block diagram showing an internal configuration of thecontrol device 300. As shown in FIG. 2, the control device 300 includesa CPU 310 and runs a predetermined program to thereby implementpredetermined functions. Instead of the CPU 310, another type ofgeneral-purpose processor, e.g. an MPU may be used. Alternatively,instead of the CPU 310, a processor dedicatedly designed so as toimplement predetermined functions may be used. That is, the controldevice 300 can include various processors (semiconductor devices) suchas CPU, MPU, FPGA, DSP, and ASIC.

Furthermore, the control device 300 includes a memory 320 and first tothird interfaces 330 to 350. The memory 320 is a medium that storesinformation and programs required when the CPU 310 executescalculations, and is properly accessed from the CPU 310.

The first interface 330 is a communication circuit through which thecontrol device 300 interchanges data and commands with the imageprojection apparatus 200, The control device 300 sends a control commandto the infrared excitation light source 230 via the first interface 330.The control device 300 receives an infrared image via the firstinterface 330 from the infrared camera 210. The control device 300 sendsa control signal via the first interface 300 to the projector 220.

The second interface 340 is a communication circuit through which thecontrol device 300 sends a video signal to the display 160. The thirdinterface 350 is a communication circuit through which the controldevice 300 receives an operation signal from the operating part 170. Thefirst to the third interfaces 330 to 350 perform interchange of data andcommands in accordance with a prescribed communication standard (e.g.USB standard, HDMI (registered trademark) standard).

Referring back to FIG. 1, the infrared excitation light source 230 is alight source that irradiates infrared excitation light 231 at least ofthe order of 800 nm that is an ICG excitation wavelength. The infraredexcitation light source 230 includes a light-emitting element (e.g. LED)that emits infrared light around 800 nm. The infrared excitation lightsource 230 can switch irradiation ON/OFF of the infrared excitationlight 231 in accordance with a control signal from the control device300. In order to reduce irradiation unevenness of the infraredexcitation light 231, it is preferred that the infrared excitation lightsource 230 be disposed right above the patient 120 lying on the surgicaltable 110.

The display 160 can display, for example, an image (hereinafter,referred to as “fluorescence image”) of the detected ICG light-emittingregion, and a menu for performing various settings in a projectionaction of the control device 300.

The operator 140 can perform various settings in the projection actionof the control device 300 by operating the operating part 170 whilelooking at the menu and icons appearing on the display 160. For example,the projection mode of the projector 220 can be set. It is also possibleto perform settings of color (hue) and number of gradations of lightprojected on the ICG light-emitting region (affected area 130). It isalso possible to perform switch settings of color and number ofgradations of light projected on the surroundings (background) andcontours of the ICG light-emitting region. The control device 300accepts a setting operation from the operator 140 to perform aprojection action depending on the operation setting.

The display 160 can be configured from an LCD display, an organic ELdisplay, etc. The operating part 170 is an input device, such as akeyboard, a mouse, a touch panel, and a stylus pen, with which theoperator 140 performs instructions and settings.

Details of parts of the image projection apparatus 200 will hereinafterbe described.

The infrared camera 210 is a camera having spectral sensitivitycharacteristics in the infrared region. The surgery support system 100needs to detect the infrared fluorescent light 223 around 850 nm emittedfrom ICG of the patient 130. To that end, the infrared camera 210 hasspectral sensitivity characteristics for the infrared region at least ofthe order of 850 nm. In order to avoid receiving light other than theinfrared fluorescent light 223 emitted from ICG, a bandpass filterallowing only light of a wavelength around 850 nm to pass therethroughmay be disposed in front of the infrared camera 210. The infrared camera210 transmits an infrared image obtained by imaging to the controldevice 300.

The projector 220 is a projection device that projects a visible-lightimage in accordance with a control signal from the control device 300,The projector 220 may irradiate light of any wavelength (color) as longas it is light within the visible-light region visible by humans. Theprojector 220 is configured to irradiate light of a plurality ofwavelengths (colors) switchable in accordance with a control signal fromthe control device 300. The projector 220 irradiates the visibleirradiation light 222 toward the dichroic mirror 211.

The dichroic mirror 211 is disposed facing each of the infrared camera210 and the projector 220. The dichroic mirror 211 is an optical elementhaving a function of reflecting light of a specific wavelength buttransmitting light of other wavelengths. In the present disclosure, aprojection port of the projector 220 is arranged in the horizontaldirection of the dichroic mirror 211, while the infrared camera 210 isarranged vertically above the dichroic mirror 211. The dichroic mirror211 has an optical characteristic of reflecting visible irradiationlight 222 irradiated from the projector 220 but transmitting infraredfluorescent light 223 heading toward an imaging surface of the infraredcamera 210. As shown in FIG. 1, the visible irradiation light 222reflected by the dichroic mirror 211 and the infrared fluorescent light223 incident on the imaging surface of the infrared camera 210 have thesame optical path. This can enhance the accuracy of irradiation of thevisible irradiation light 222 onto the region (affected area 130)issuing the infrared fluorescent light 223.

In the above, the infrared camera 210 is an example of a detector of thepresent disclosure. The projector 220 is an example of a projector thatperforms projection using visible light of the present disclosure. Thecontrol device 300 is an example of a controller of the presentdisclosure. And, the surgery support system 100 is an example of theimage projection apparatus of the present disclosure.

3. Actions of Surgery Support System

FIG. 3 is a flowchart showing actions of the surgery support system 100.Referring to the flowchart of FIG. 3, actions of the surgery supportsystem 100 will be described.

The infrared excitation light source 230 irradiates infrared light ontothe affected area 130 of the patient 120 (Step 10). ICG accumulated inthe affected area 130 emits infrared fluorescent light when receivinginfrared light. The infrared camera 210 images a region including theaffected area 130 to generate an infrared image.

Via the first interface 330, the control device 300 acquires theinfrared image imaged and generated by the infrared camera 210 (Step11). From the infrared image, the control device 300 (i.e. CPU 310)generates a display image for display on the display 160 (Step 12). Thatis, the control device 300 generates the display image, based on theemission intensity of the ICG light-emitting region in the acquiredinfrared image (i.e. the pixel signal strength of the infrared image)(Step 12).

Furthermore, from the infrared image, the control device 300 generates aprojection image for projection onto the ICG light-emitting region (Step13). Generation processes of the display image and the projection imagewill be described later.

And, the control device 300 sends the generated display image to thedisplay 160, allowing display on the display 160 (Step 14). The controldevice 300 sends data of the generated projection image to the imageprojection apparatus 200, allowing the projector 220 to project theprojection image with visible light (Step 15).

Until accepting an ending operation related to the projection actions ofthe surgery support system 100 (Step 16), the control device 300executes the above processes (Step 10 to Step 15) repeatedly. Thisallows visible-light projection image to be projected onto the ICGlight-emitting region of the affected area 130, enabling the doctor tovisually easily recognize a medical treatment area, etc.

3.1 Color Control of Display Image and Projection Image

Description will be given of color control on the display image or theprojection image by the surgery support system 100.

The surgery support system 100 has one display mode as an action mode ofthe display 160. The surgery support system 100 has first to thirdprojection modes as projection modes of the projector 220. Although adisplay image or a projection image is generated from an infrared imagein each mode, the color conversion processing manner at that timediffers by mode.

FIG. 4 is a view explaining color control of the display image and theprojection image in the display mode and the first to third projectionmodes in the surgery support system 100. In a table shown in FIG. 4, afirst stage explains control on color of the display image in thedisplay mode, while second to fourth stages explain control on color ofthe projection image in each projection mode.

FIG. 5 is a view explaining a general HSV color space. The HSV colorspace expresses colors by Hue (H), Saturation (S), and Value (V). Hue(H) represents a color type such as red, blue, and yellow, except whiteand black. Saturation (S) is color vividness and takes a value from 0 to100%. Value (V) is color brightness and takes a value from 0 to 100%.White is when the saturation is 0% with the value being 100%, and blackis when the saturation is 0% with the value being 0%.

In the following description, H0 means that the hue is 0 degree, H120means that the hue is 120 degrees, S0 means that the saturation is 0%,S100 means that the saturation is 100%, V0 means that the value is 0%,and V100 means that the value is 100%, and the same applies to the othernotations. The subsequent HSV numerical values are one example, and e.g.any numerical values other than H0 may be employed as long as theoperator is recognizable as red. Furthermore, color may be set using acolor space, such as RGB color segment, other than the HSV color space.

Hereinafter, referring to FIG. 4, color control in the display mode andthe first to third projection modes will be described.

(1) Display Mode

A display image appearing on the display 160 is generated bycolor-converting an infrared image (esp. a fluorescence image includingthe ICG light-emitting region) received from the infrared camera 210.The display image is displayed in black and white. That is, the displayimage is generated by color-converting the color of each pixel of theinfrared image as follows. Specifically, for each pixel of the infraredimage, the saturation (S) is fixed at 0% (S0) (minimum) while the value(V) is set within a range from 0 to 100% (V0 to V100) depending on thebrightness (fluorescence emission signal intensity) of each pixel of theinfrared image. The value (V) of the display image is set to a highervalue according as the infrared image has a stronger brightness. As tothe hue, any value is acceptable since the saturation is 0%. Thegradation of the display image is set to the maximum gradation (1024gradations in this embodiment).

(2) Projection Mode

A projection image projected by the projector 220 is generated bycolor-converting an infrared image (esp. a fluorescence image) receivedfrom the infrared camera 210. Note that in the projection mode, thebackground other than the affected area in the infrared image isdisplayed in white so as to fulfill the function as lighting.

a) First Projection Mode

A first projection mode is a mode for generating a projection image bycolor-converting an infrared image in full gradation (1024 gradations).In the first projection mode, the hue of each pixel of the projectionimage is fixed at a hue (e.g. H240 in FIG. 4) selected by the operatorand the value is fixed at 100%. The saturation (S) of each pixel of theprojection image is set depending on the brightness, i.e. ICGfluorescence emission intensity, of the infrared image. In the exampleof FIG. 4, color (color visually recognized by the operator) of eachpixel of the projection image turns from white into blue according asthe ICG fluorescence emission intensity becomes stronger.

b) Second Projection Mode

A second projection mode is a mode in which the number of gradations inthe first projection mode is reduced and is a mode displaying, inmonocolor, a projection image obtained by mufti-valuing the infraredimage. In the second projection mode, a monocolored projection image isgenerated by binarizing, quaternarizing, or octalizing the infraredimage based on the pixel brightness. The gradation of the projectionimage is previously set in the control device 300 by the operator 140via the operating part 170. Similar to the first projection mode, in thesecond projection mode, the hue (H) is fixed at a hue (e.g. H240 in FIG.4) previously selected by the operator, the value (V) is fixed at 100%,and the saturation (S) is set depending on the brightness (ICGfluorescence emission intensity) of the infrared image.

For example, in the case of displaying a 2-valued image, the hue (H) andthe value (V) are fixed and the saturation (S) is set to S0 or 3100depending on the brightness of the infrared image. Or, in the case ofdisplaying a 4-valued image, the saturation (3) is set to any one of S0,S33, S67, and 3100 depending on the brightness of the infrared image. Byvirtue of such a second projection mode, the visibility at the boundaryof the projection image is improved.

c) Third Projection Mode

A third projection mode is a mode in which a projection image obtainedby multi-valuing an infrared image is displayed in multicolor. In thethird projection mode, a multi-colored projection image is generated bymulti-valuing the infrared image based on the pixel brightness. In thethird projection mode, the hue (H) is set to any one of 6 types (−H0,H30, H60, H120, H240) depending on the brightness of the infrared image.The saturation (S) is set to any value of 6 steps (S0, S20, S40, S60,S80, S100) depending on the brightness of the infrared image. The value(V) is also varied depending on the brightness of the infrared image. Inthe third projection mode, an image is displayed using different colorsthat depend on the ICG fluorescence intensity, so that an image coloreddifferently for each distribution of the ICG fluorescence intensity isprojected onto the ICG light-emitting region of the affected area 130.This leads to an improved visibility of the ICG concentrationdistribution in the affected area 130.

Note that, since white appears irrespective of the value of the hue ifthe saturation is 0% and the value is 100%, - (any value) is given to aportion having the lowest ICG light emission intensity. Moreover, if theICG fluorescence emission intensity is less than a predetermined value,white is given that is the same color as the background having the ICGfluorescence emission intensity of 0.

The color (color visually recognized by the operator) of each pixel of aprojection image is displayed in 6 stages of white, red, orange, yellow,green, and blue according as the ICG fluorescence emission intensityincreases.

Although the value may be constant, generally, orange and yellow have alower visibility according as the value becomes higher, so that they aregiven a little lower value as compared with the other colors, to improvethe visibility.

FIG. 6 is a view showing, in contrast, examples of an image displayed inthe display mode and of images projected in the first to thirdprojection modes. FIG. 6(A) shows an example of a black and white imagein full gradations (1024 gradations), displayed on the display 160 inthe display mode. FIG. 6(B) shows an example of a monocolored image infull gradation, projected from the projector 220 in the first projectionmode. FIG. 6(C) shows examples of a monocolored multi-valued image,projected from the projector 220 in the second projection mode. FIG.6(C) shows a 2-valued image, a 4-valued image, and an 8-valued image,individually. FIG. 6(D) shows an example of a multi-colored multi-valuedimage, projected from the projector 220 in the third projection mode.

FIG. 7A is a view explaining an image of an infrared image imaged by theinfrared camera. In FIG. 7A, there is an incision 52 in a body surface50 of the patient 120. ICG is accumulated in a region R1 a in aninterior 54 of the patient body, visible from the incision 52. A regionRib around an ICG injection part 56 is also an ICG accumulated portion.In affected areas A to E within these regions R1 a and R1 b, ICG isaccumulated in respective different concentrations. In this case, theaffected areas E to A have an ICG concentration lowered in the mentionedorder. Thus, the affected area E has a strongest ECG fluorescenceemission intensity, and the affected areas D, C, B, and A have an ICGfluorescence emission intensity weakened in the mentioned order. Fromthe infrared image, the CPU 310 detects the regions Ria and Rib (anexample of a first region) where fluorescence is seen, to color-convertpixels of these regions R1 a and R1 b in the infrared image to generatea projection image,

FIG. 7B is a view showing an image of a projection image generated fromthe infrared image shown in FIG. 7A in the third projection mode. Anoverall region R2 (an example of a second region) of the projectionimage shown in FIG. 7B includes the detected regions R1 a and R1 b and abackground region R3 thereof. In the projection image, pixels in theportion of the affected area E having a strongest ICG fluorescenceemission intensity are set to blue in hue and to S100 in saturation.Pixels in the portion of the affected area D having a next strongest ICGfluorescence emission intensity are set to green in hue and to S80 insaturation. Pixels in the portion of the affected area C having afurther next strongest ICG fluorescence emission intensity are set toyellow in hue and to S60 in saturation. Hereinafter, similarly, pixelsin the portion of the affected area B are set to orange in hue and toS40 in saturation. Pixels in the portion of the affected area A are setto red in hue and to S20 in saturation. In this manner, respectivelydifferently colored projection images are generated for the affectedareas A to E. The background region R3 is white. As shown in 7B, imagescolored differently depending on the ICG fluorescence intensitydistribution are projected, with the result that the intensitydistribution is visually recognized more easily.

By allowing the ICG fluorescence intensity distribution to be visuallyrecognized, it becomes possible to easily confirm the range of theaffected area 130 or the depth of a site where the affected area 130exists in an organ. For example, in the case where the affected area 130is deeply buried in the organ, ICG fluorescence emission detected on thebody surface 50 is weak, so that it is projected as a red spot on thebody surface 50. As this region projected in red is incised, theaffected area 130 becomes gradually exposed, with the result that ICGfluorescence emission intensity becomes stronger, allowing theprojection image to turn to orange, yellow, green, and blue. Finally, asshown in FIG. 7B, the affected area 130 is projected as a portion of theregion R1 a, so that if the affected area 130 needs to be excised, itbecomes possible to easily recognize and excise a region projected inblue as the affected area 130 requiring excision.

<Hue Range in Third Projection Mode>

FIG. 8 is a view explaining the range of hue of projection image colorsset in the third projection mode. FIG. 8 shows color allocation for ICGfluorescence emission signal intensity in the case where the projectionimage is represented in 8 gradations (8-valued) and in the case wherethe projection image is represented in 7 gradations (7-valued).

In a surgical field, there exist blood, body surface, fat, lymph node,etc. as the background of the projection image. Blood is red, bodysurface is color of skin (skin color such as pale orange in thisembodiment), fat is yellow, and lymph node is yellow to translucent, andhence the background color of the surgical field becomes warm colors. Inthis embodiment, to facilitate visual distinction of the projectionimage from its background, a color (i.e. cold color) having a largedifference in value and hue from the surgical field background color isused as a color allocated to a region having a relatively strong ICGfluorescence emission intensity. On the other hand, a color (warm color)having a small difference in value and hue from the surgical fieldbackground color (human body color) is used as a color allocated to aregion having a relatively weak ICG fluorescence emission intensity.

For example, a region having a strong ICG fluorescence emissionintensity is represented by a cold color and, according as the intensitybecomes weaker, the color is varied to a warmer color. Here, in the caseof representing the physical quantity distribution in multicolor likethermography, it is conventionally general to represent a region havinga strong intensity by a warm color (red) and to represent a regionhaving a weak intensity by a cold color (blue). In the case of a medicalsystem like this embodiment, however, blood is “red”, fat is “yellow”,and lymph node is “yellow to translucent”, and therefore if a regionhaving a strong intensity is represented by a warm color(red) there is aproblem that it is difficult to visually distinguish the projectionimage from blood, fat, lymph node, etc. Thus, in this embodiment, theregion having a strong ICG fluorescence emission intensity isrepresented by a cold color whereas the region having a weak intensityis represented by a warm color.

For this end, the hue of a color allocated to a region (pixels) having astrongest ICG fluorescence emission intensity is set to a value withinthe range apart ±60 degrees or more from the range (0 degree or more and60 degrees or less) of the hue of the surgical field background color.More specifically, the color allocated to a region (pixels) having astrongest ICG fluorescence emission intensity (signal intensity=1024) isset to a color whose hue has a value within the range of 120 degrees ormore and 300 degrees or less. This allows the operator (doctor, etc.) toeasily visually distinguish the projection image from the backgroundsuch as blood and body surface.

For example, if the projection color in the case of a high ICGfluorescence intensity is a warm color such as red and orange, the colorbecomes a color similar to the surroundings of the affected area 130,rendering visual recognition of the excision region difficult. Inparticular, if the surgery region is a minute region inside an organ,even though the minute region to be excised has been found, the visualrecognition of the excision region may become impossible due tounexpected bleeding, etc. On the contrary, if the projection color inthe case of a high ICG fluorescence emission intensity is a cold colorlike blue, even though unexpected bleeding has occurred when excisingthe minute region, color distinction from blood is clear and hence thevisual recognition of the excision region is not hindered, making aprecise excision possible.

Alternatively, depending on the hue of an organ where the affected area130 exists, for example, if the color of the organ is a cold color,setting may be made such that the color changes stepwise from a warmcolor toward a cold color according as the emission intensity decreasesfrom the highest value. By setting in this manner, the visibility of theaffected area 130 projected in a warm color can be enhanced in a coldcolor of the organ.

With regard to color allocated to a region (pixels) having a signalintensity of 0 or more and 128 or less, the saturation is set to 0% andthe value is set to 100%. In other words, the color is set to the samecolor as the background, i.e. to white. This enables the influence ofnoise of a weak emission intensity to be eliminated. Moreover, alighting function by the projection image can be implemented,

3.2 Display Image Generation Process

A display image generation process (Step 12 of FIG. 3) will bedescribed. For each of pixels of an infrared image, the CPU 310 sets thehue to any value and sets the saturation to 0%. Furthermore, the CPU 310sets the value of each pixel of the infrared image depending on the ICGfluorescence emission intensity of each pixel. Here, the value is variedin 1024 gradations depending on the emission intensity. As a result, adisplay image of full gradation (1024 gradations) having brightness thatdepends on the fluorescence emission intensity of the affected area 130is generated and is displayed on the display 160.

3.3 Projection Image Generation Process

Details (Step 13 of FIG. 13) of a projection image generation processwill be described. FIG. 9 is a flowchart showing the projection imagegeneration process.

In FIG. 9, the CPU 310 first determines the type of the projection modebeing currently set (Step 21).

If the currently set mode is the first projection mode, the CPU 310performs color conversion processing in accordance with the firstprojection mode (Step 22). Specifically, for each pixel of an infraredimage, the CPU 310 sets the hue to a value selected by the operator andsets the value to 100% (V100). For example, in the example of FIG. 4,the hue is fixed at H240. Furthermore, the CPU 310 sets the saturationof each pixel of the infrared image, depending on the ICG fluorescenceemission intensity of each pixel. Here, the saturation is varied in 1024gradations depending on the emission intensity. Consequently, afull-gradation projection image is generated.

If the currently set mode is the second projection mode, the CPU 310performs multi-valued processing in accordance with the secondprojection mode (Step 23). That is, the original infrared image issubjected to multi-valued processing in gradations such as 2 gradations,4 gradations, and 8 gradations, set by the operator 140.

For example, in the case of being multi-valued in 2 gradations when thegradation of the original infrared image is 1024 gradations, a thresholdin the multi-valued processing is set to 511. And, in the multi-valuedprocessing, as to pixels having gradations of 0 to 511 in the infraredimage, their pixel values are converted to 0. On the contrary, as topixels having gradations of 512 to 1023 in the infrared image, theirpixel values are converted to 1023. By such multi-valued processing,2-gradation image is generated from the infrared image.

In the case of being multi-valued in 4 gradations, three thresholds of255, 511, and 767 are set. And, as to pixels having gradations of 0 to255 in the infrared image, their pixel values are converted to 0. As topixels having gradations of 256 to 511 in the infrared image, theirpixel values are converted to 341. For pixels having gradations of 512to 767 in the infrared image, their pixel values are converted to 682.For pixels having gradations of 768 to 1023 in the infrared image, theirpixel values are converted to 1023. By such multi-valued processing,4-gradation image is generated from the infrared image.

After the multi-valued processing (Step 23), the CPU 310 performs colorconversion processing in accordance with the second projection mode(Step 24). Specifically, for each pixel of the multi-valued image, theCPU 310 fixes the hue at a value selected by the operator, varies thesaturation, and fixes the value to 100%. For instance, in the example ofFIG. 4, the hue is fixed at H240. Furthermore, the CPU 310 sets thesaturation of each pixel, depending on the pixel value of each pixel(i.e. the ICG fluorescence emission intensity of each pixel) of themulti-valued image. For example, as shown in FIG. 4, in the case ofbeing set to 2 gradations, the saturation of a pixel having a pixelvalue of 0 is set to S0, while the saturation of a pixel having a pixelvalue of 1023 is set to S100. In the case of being set to 4 gradations,the saturation is set to any one of S0, S33, S67, and S100 depending onthe pixel value of each pixel of the multi-valued image. In the case ofbeing set to 8 gradations, the saturation of each pixel is set to anyone of S0, S14, S29, 543, S57, S71, S86, and S100 depending on the pixelvalue.

If the currently set mode is the third projection mode, the infraredimage is multi-valued in predetermined gradations (e.g. 6, 7, or 8gradations) set in advance for the third projection mode (Step 25). Forinstance, in the example shown in FIG. 4, it is multi-valued in 6gradations.

Afterward, the CPU 310 performs color conversion processing inaccordance with the third projection mode (Step 26). Specifically, foreach pixel of the multi-valued image, the CPU 310 sets the hue, value,and saturation depending on the pixel value of each pixel. For instance,in the example shown in FIG. 4, the hue is varied in the order of −, H0,H30, H60, H120, and H240 according as the pixel value (ICG fluorescenceemission intensity) increases. At that time, the saturation and thevalue are also varied similarly stepwise. That is, when viewed from theoperator, variation occurs in the order of white, red, orange, yellow,green, and blue. Thus, the concentration distribution of ICC accumulatedin the affected area 130 in the surgical field is displayed in color,enabling the doctor, etc. performing surgery to easily recognize theconcentration distribution of ICG accumulated in the affected area 130.

3.4 Black-and-White Inversion Function

The surgery support system 100 further has a function of inverting blackand white in the display mode and the projection modes. That is, when ablack-and-white inversion instruction is issued via the operating part170 by the operator 140, black and white are inverted in the displayimage and the projection image.

FIG. 10 is a view explaining color conversion processing(black-and-white inverted version) in the case where black and white areinverted in the color conversion processing shown in FIG. 4. Forinstance, in the display mode, the example of FIG. 4 varied the colorfrom black to white, in other words, increased the value, according asthe fluorescence emission intensity increases. On the contrary, theexample of FIG. 10 varies the color from white to black, that is,decreases the value, according as the fluorescence emission intensityincreases. In the first projection mode, the example of FIG. 4 variedthe saturation so that variation from “white” to “blue” occurs accordingas the emission intensity increases, with the value being fixed at 100%.On the contrary, the example of FIG. 10 varies the value so thatvariation from “black” to “blue” occurs according as the emissionintensity increases, with the saturation being fixed at 100%.

In accordance with a black-and-white inversion instruction from theoperator, the CPU 310 switches the color conversion processing to beexecuted between the color conversion processing shown in FIG. 4 and thecolor conversion processing shown in FIG. 10, thereby enablingblack-and-white inversion in the display mode and the projection modes.

4. Effects, Etc.

As described above, the surgery support system 100 of this embodimentcomprises the infrared camera 210 that detects a first region emittingnon-visible light, the projector 220 that projects a visible-light imageonto a second region (e.g. region R2) including the first region (e.g.regions R1 a and R1 b) detected by the infrared camera 210, and thecontrol device 30 (or the CPU 310) that generates the visible-lightimage. The control device 300 varies in a stepwise manner the hues ofpixels of the visible-light image, depending on the non-visible lightemission intensities at positions corresponding to that pixels withinthe first region. At that time, the control device 300 sets the hue sothat the colors of the pixels of the visible-light image vary stepwisefrom cold colors to warm colors according as the emission intensitiesbecome lower from the highest value.

This configuration allows a visible-light image to be projected inmulticolor onto the first region where fluorescence emission has beendetected, depending on the ICG fluorescence emission intensitydistribution. In particular, use of cold colors in a strong emissionintensity region facilitates distinction from warm-colored backgroundseen in the surgical field, improving the visibility of the projectedimage.

The control device 300 may set the hue of color of pixels correspondingto a highest emission intensity region, to within a range of 120 degreesor more and 300 degrees or less in HSV color space. For example, colorof the visible-light image pixels corresponding to the highest emissionintensity region may be set to blue or green. The background color inthe surgical field is a warm color and its hue range is 0 degree to 60degrees. Thus, by setting as the above the hue of pixel colorcorresponding to the highest emission intensity region, it is separatedapart 60 degrees or more from the hue of the background color, resultingin an improved visibility of the projected image.

The control device 300 may set stepwise at least one of the saturationand the value of visible-light image pixels, depending on thenon-visible-light emission intensities at positions corresponding tothat pixels within the first region (see Third Projection Mode of FIG.4). By making the saturation and the value together with the huevariable depending on the emission intensities, the projected imagevisibility can further be improved.

The control device 300 may set the settable lowest step color to white.This enables the influence of noise of a weak emission intensity to beeliminated due to the same color as the background color. Moreover, thelighting function by the projection image can be implemented.

The surgery support system 100 may further include the operating part170 for the operator 140 performing various settings. The control device300 may include, as the action modes of the projector 220, a pluralityof projection modes switchable in accordance with an operation on theoperating part 170. The plurality of projection modes include at yeastone of the first projection mode and the second projection mode, and thethird projection mode. The first projection mode is a mode in which thesaturations of a visible-light image pixels are continuously varieddepending on the non-visible-light emission intensities at positionscorresponding to that pixels within the first region. The secondprojection mode is a mode in which the saturations of visible-lightimage pixels are stepwise varied depending on the non-visible-lightemission intensities at positions corresponding to that pixels withinthe first region. The third projection mode is a mode in which the huesof visible-light image pixels are varied depending on thenon-visible-light emission intensities at positions corresponding tothat pixels within the first region. By providing a plurality ofswitchable projection modes, a projection image suitable for thecontents or the situation of surgery can be selected.

Other Embodiments

As above, a first embodiment has been described as an exemplification oftechnique disclosed in the present application. However, the techniqueof the present disclosure is not limited thereto and is applicable toany embodiments in which changes, permutations, additions, omissions,etc. have properly been made. It is also possible to combine theconstituent elements described in the above first embodiment, to providea new embodiment. Thus, other embodiments will hereinbelow beexemplified.

In the first embodiment, the operator 140 performed various settings forthe surgery support system 100 by using and operating the operating part170 while seeing the display on the display 160. The present disclosureis not limited thereto. Information required for setting may beannounced from a speaker connected to the control device 300. And, afterhearing that announcement, the operator 140 may speak desired settinginto a microphone, to thereby perform a setting operation. Naturally,while seeing buttons on a menu appearing on the display 160, theoperator 140 may speak desired setting into the microphone, to therebyperform a selecting operation.

The number of gradations shown in the first embodiment is an example andis properly set depending on the type of the affected area, the case,etc.

Although in the first embodiment the device supporting surgery has beendescribed by way of example, the image projection system of the presentdisclosure is not limited thereto. For example, also in the case wherework needs to be done for objects whose change in state cannot bevisually confirmed in e.g. a construction site, a mining site, abuilding site, or a factory that processes materials, the idea of theimage projection system of the present disclosure is applicable. In thiscase, the intensity distribution colors may be set according to thesituation at the above site.

In place of the infrared fluorescent light from ICG of the firstembodiment, a fluorescent material may be applied to, kneaded in, orpoured into the objects whose change in state cannot be visuallyconfirmed in e.g. a construction site, a mining site, a building site,or a factory that processes materials, to obtain a target to be imagedby the infrared camera 210, onto which a visible-light image isprojected. Instead of light emission, a heating portion may be detectedby a heat sensor so that a visible-light image is projected onto theheating portion. In this case, a far-infrared ray issued from theheating portion is an example of non-visible light in the presentdisclosure.

As above, the embodiments have been described as exemplifications of thetechnique in the present disclosure. To that end, the accompanyingdrawings and detailed description have been provided.

Accordingly, the constituent elements described in the accompanyingdrawings and detailed description may encompass not only essentialcomponents for solving the problems but also non-essential componentsfor solving the problems, for the exemplifying purposes only. Therefore,immediately from the fact that those non-essential components aredescribed in the accompanying drawings and detailed description, thosenon-essential components should not be construed as essential.

Since the above embodiments are for the purpose of exemplifying thetechnique in the present disclosure, various changes, permutations,additions, omissions, etc. can be made without departing from patentclaims or their equivalent scope.

The image projection system in the present disclosure is not limited tosurgical applications but is applicable to situations where work is donefor objects whose change in state cannot be visually confirmed in e.g. aconstruction site, a mining site, a building site, or a factory thatprocesses materials.

What is claimed is:
 1. An image projection apparatus comprising: adetector that detects a first region emitting non-visible light; aprojector that projects a visible-light image onto a second regionincluding the first region detected by the detector; and a controllerconfigured to generate the visible-light image so as to vary colors ofpixels of the visible-light image in a stepwise manner depending onemission intensities of the non-visible light at positions correspondingto the pixels within the first region.
 2. The image projection apparatusaccording to claim 1, wherein the controller sets the colors of thepixels of the visible-light image so as to vary the colors in thestepwise manner from cold colors to warm colors according as theemission intensities become lower from a highest value.
 3. The imageprojection apparatus according to claim 1, wherein the controller setsthe colors of the pixels corresponding to a region having the highestemission intensity to within a range of 120 degrees or more and 300degrees or less in hue in HSV color space.
 4. The image projectionapparatus according to claim 2, wherein the controller sets the colorsof the pixels of the visible-light image corresponding to a regionhaving the highest emission intensity to blue or green.
 5. The imageprojection apparatus according to claim 1, wherein the controller setsat least one of saturation and value of the pixels of the visible-lightimage in HSV color space in the stepwise manner depending on theemission intensity of the non-visible light at the positionscorresponding to the pixels within the first region.
 6. The imageprojection apparatus according to claim 1, wherein the controller setsthe color at a settable lowest step to white.
 7. The image projectionapparatus according to claim 1, further comprising an operating part foran operator performing various settings, wherein the controller isconfigured to control the projector according to an action modeincluding a plurality of projection modes switchable in accordance withan operation on the operating part, the plurality of projection modesincluding at least one of a first projection mode and a secondprojection mode, and a third projection mode, the first projection modeis a mode in which saturations of the pixels of the visible-light imageare varied continuously depending on the emission intensities of thenon-visible light at the positions corresponding to the pixels withinthe first region, the second projection mode is a mode in whichsaturations of the pixels of the visible-light image are varied in astepwise manner depending on the emission intensities of the non-visiblelight at the positions corresponding to the pixels within the firstregion, and the third projection mode is a mode in which hues of thepixels of the visible-light image are varied depending on the emissionintensities of the non-visible light at the positions corresponding tothe pixel within the first region.